PATENT COOPERATION TREATY 



RK'D 0 7 JUL 2005 



PCT 

INTERNATIONAL PRELIMINARY REPORT ON PAtS^cBHtItF'^ — 

(Chapter II of the. Patent Cooperation Treaty) 

(PCT Article 36 and Rule 70) 



Applicant's or agenf s file reference 
Q080PCT 



FOR FURTHER ACTION 



See Form PCT^PEA/416 



International application No. 
PCTyEP2004y007098 



International filing date (dayAnonUvyear) 
30.06.2004 



Prtortty date (dayJhnonth^fear) 
01.07.2003 



International Patent Classification (IPC) or national ciassiflcation and IPC 
G06T3yD0 



Applicant 

THOMSON LICENSING SA 



1. This report is the International preliminary examination report, established by this International Preliminary Examining 
Authority under Article 35 and transmitted to the applicant according to Article 36. 

2. This REPORT consists of a total of ^sheets, including this cover sheet. 

3. This report Is also accompanied by ANNEXES, comprising: 

a. SI sent to the applicant and to the International Bureau) a total of 1 0 sheets, as follows: 

□ sheets of the description, claims andA^r drawings which have been amended and are the basis of this report 
andA^r sheets containing rectifications authorized by this Authority (see Rule 70.16 and Section 607 of the 
Administrative Instructions). 

□ sheets which supersede earlier sheets, but which this Authority considers contain an amendment that goes 
beyond the disclosure In the international application as filed, as indicated in item 4 of Box No. I and the 
Supplemental Box. 

b. □ (sent to the International Bureau only) a total of (indicate type and number of electronic carrier(s)) , containing a 

sequence listing and>br tables related thereto, in computer readable form only, as indicated in the Supplemental 
Box Relating to Sequence Usting (see Section 802 of the Administrative Instructions). 



4. This report contains indications relating to the following items: 





Box 


No. 


1 


□ 


Box 


No. 


II 


□ 


Box 


No. 


111 


□ 


Box 


No. 


IV 


la 


Box 


No. 


V 


□ 


Box 


No. 


VI 


□ 


Box 


No. 


VII 


□ 


Box 


No. 


VIII 



Basis of the opinion 
Priority 

Non-establishment of opinion with regard to novelty, inventive step and industrial applicability 
Lack of unity of invention 

Reasoned statement under Article 35(2) with regard to novelty, inventive step or industrial 
applicability; citations and explanations supporting such statement 

Certain documents cited 

Certain defects in the International application 

Certain observations on the international application 



Date of submission of the demand 



18.02.2005 



Date of completion of this report 



08.07.2005 



Name and mailing address of the international 
preliminary examining authority: 

European Patent Office - P.B. 5818 Patentlaan 2 
NL-2280 HV Rijswijk - Pays Bas 
Tel. +31 70 340 - 2040 Tx: 31 651 epo nl 
Fax: +31 70 340-3016 



Authorized Officer 
Deltorn, J-M 

Telephone No. +31 70 340-3468 



(J>) 



Fomi PCT/IPEA/409 (Cover Sheet) (January 2004) 



INTERNATIONAL PRELIMINARY REPORT 
ON PATENTABILITY 



International application No. 
PCTyEP2OO4/0O7O98 



Box No. I Basis of the report 



1 . With regard to the language, this report is based on the international application in the language in which it was 
filed, unless othenwise indicated under this item. 

□ This report is based on translations from the original language into the following language , 
which is the language of a translation fumished for the purposes of: 

□ international search (under Rules 12.3 and 23.1 (b)) 

□ publication of the international application (under Rule 12.4) 

□ international preliminary examination (under Rules 55.2 and/or 55.3) 

2. With regard to the elements'" of the international application, this report is based on (replacement sheets which 
have been fumished to the receiving Office in response to an invitation under Article 14 are referred to in this 
report as "originally filed" and are not annexed to this report): 

Description, Pages 

1 -3, 5-24, 26-28 as originally filed 

4, 4a, 25 received on 22.02.2005 with letter of 1 4.02.2005 
Claims, Numbers 

1-14 received on 22.02.2005 with letter of 1 4.02.2005 
Drawings, Sheets 

1/1 as originally filed 

□ a sequence listing andybr any related table(s) - see Supplemental Box Relating to Sequence Listing 

3. □ The amendments have resulted in the cancellation of: 

□ the description, pages 

□ the claims, Nos. 

□ the drawings, sheetsyfigs 

□ the sequence listing (specify): 

□ any table(s) related to sequence listing (specify): 

4. □ This report has been established as if (some of) the amendments annexed to this report and listed below 
had not been made, since they have been considered to go beyond the disclosure as filed, as indicated in the 
Supplemental Box (Rule 70.2(c)). 

□ the description, pages 

□ the claims, Nos. 

□ the drawings, sheets/figs 

□ the sequence listing (specify): 

□ any table(s) related to sequence listing (specify): 

* If item 4 applies, some or all of these sheets may be marked "superseded. " 
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Box No. V Reasoned statement under Article 35(2) with regard to novelty, inventive step or industrial 
applicability; citations and explanations supporting such statement 

1. Statement 

Novelty (N) Yes: Claims 1-14 

No: Claims 

Inventive step (IS) Yes: Claims 1-14 

No: Claims 

Industrial applicability (lA) Yes: Claims 1-14 

No: Claims 

2. Citations and explanations (Rule 70.7): 
see separate sheet 
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Re Item V 

Reasoned statement with regard to novelty, inventive step or industrial 
applicability; citations and explanations supporting such statement 

1. Reference is made to the following document: 

D1 : COLLINS R T ET AL: "Matching perspective views of coplanar structures 
using projective unwarping and similarity matching" COMPUTER VISION 
AND PATTERN RECOGNITION, 1993. PROCEEDINGS CVPR '93, 1993 
IEEE COMPUTER SOCIETY CONFERENCE ON NEW YORK, NY, USA 15- 
17 JUNE 1993, LOS ALAMITOS, CA, USA, IEEE COMPUT. SOC, 15 June 
1993, pages 240-245 



2. CLAIM 1 

2.1 Novelty 

The document D1 is regarded as being the closest prior art to the subject-matter 
of claim 1 and shows (the references in parentheses applying to this document): 

An automatic resetting method using electronic means intended for a geometric 
model of a scene over a picture of a scene (page 240, section 1), 
characterized in that the electronic device calculates a fine homography function 
Hr for resetting Into two main phases: 

(i) a first preliminary phase of determination of an average resetting 
homography consisting in detennining an average homography function Hm 
applicable to the model with average adjustment over a sample of pictures of 
the scene taken previously (page 242, first paragraph of section 3), 

(ii) a second, rough resetting phase consisting after application of the average 
homography function Hm to the model in determining a rough homography 
function Hg (page 242, first paragraph of section 3). 

The subject-matter of claim 1 differs from this known method in that: 
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(a) The model and the picture of the scene being stored in the mennory of an 
electronic device in the form of pixel matrices, the scene including fixed 
references with respect to the remainder of the scene, whereas the 
references may be specifically detected within the matrices, 

(b) The picture being taken by a camera arranged in a given zone with respect 
to the ground in a location of the zone and according to a shot angle 
determined relative to the scene, the electronic means comparing the picture 
with the model having been adjusted in perspective by homography for 
superimposition of the references, 

(c) The second, rough resetting phase further Includes the steps of: 

(1) extracting reference pixels and producing a pair of binary matrices 
containing vectical and an horizontal contour points, 

(ii) for each horizontal and vertical reference matrices, computing a 
distance matrix in which the value of each element of the matrix 
corresponds to the distance to the closest reference point, according to 
the horizontal line and the vertical line respectively 

(ill) applying an average homographic function to the reference lines of the 
model and comparing the resulting matrix to the horizontal and vertical 
reference matrices; calculating a homography function by regression 
with m inimisation of the medial of the square of the distance between 
pairs of matched pixels, 

(Iv) Identifying the pairs of pixels corresponding to non aberrant matches 

(v) adjusting by least square regression over all the non aberrant pixel 
pairs the homography function In order to produce a rough 
homography. 

(d) The feature of Claim 1 includes an additional third step of refinement of the 
Homography function. 

The subject-matter of claim 1 is therefore new (Article 33(2) PCT). 

2.2 Inventive Step: 

The problem to be solved by the present invention may be regarded as: 
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How to provide an automatic resetting method between an image of a scene 
including fixed references and a geometric model of said scene 

The solution to this problem proposed in claim 1 of the present application is 
considered as Involving an inventive step (Article 33(3) PCT) for the following 
reasons: 

The incremental calculation of a fine homography function between a scene and a 
geometric model of said scene disclosed in Claim 1 provides an improved, more 
robust, resetting method that allows, in particular, to match a model on an image 
corresponding to said model seen in perspective without requirering to determine 
a vanishing point out of reference lines detected in the image. The method of 
Claim 1 is indeed based on the matching of reference points and is therefore less 
prone to being affected by the partial occlusion of reference lines by foreground 
objects in the image. Such a method is not known from nor suggested by the 
available prior art. 

3. INDEPENDENT CLAIMS 11, 13 AND 14 

The same reasoning applies, mutatis mutandis, to the corresponding independent 
device claim 11 and information storage medium including a programme claims 13 
and 14, which are therefore also considered novel and inventive (Articles 33(2) 
and 33(3) PCT). 



4. DEPENDENT CLAIMS 2-10 AND 12 

Claims 2-10 are dependent on claim 1 and as such also meet the requirements of 
the PCT with respect to novelty and inventive step. 

Claim 12 dependent on claim 12 and as such also meets the requirements of the 
PCT with respect to novelty and inventive step. 
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Deriche filtering is used. The intersections of tlie different 
straiglit lines found form a collection of particular points 
serving for resetting with respect to the theoretic modeL The 
Initial resetting is performed manually on the first picture 

S while associating 4 points identified in the picture with their 
counterpart in the theoretic model. For the following pictures, 
an algorithm for compensation of the dominant movement 
enables to track the matching points throughout the 
sequence. The shortcomings of such method are mainly the 

10 use of manual initialisation, the sensitivity of the line 
detection algorithm and the difficulty of adaptation to a more 
complex ground model which does not exhibit any equivalent 
ground marlcings. 

One also knows following documents: 

IS COLLINS R T and AL: "Matching perspective views of 

coplanar structures using projective unwarping and similarity 
matching" COMPUTER VISION AND PATTERN 
RECOGNITION. 1993. PROCEEDINGS CVPR '93. 1993 IEEE 
COMPUTER SOCIETY CONFERENCE ON NEW YORK, NY, 

20 USA 15-17 JUNE 1993. LOS ALAMITOS. CA. USA, IEEE 
COMPUT. SOC. June 15,1993, pages 240-245; 

CHETVERIKOV D., KHENOKH Y.: "Matching for shape 
defect detection" LECTURE NOTES IN COMPUTER 
SCIENCES, vol. 1689. 1999, pages 367-374, HEIDELBERG; 

25 FAUGERAS, O.: "Three dimensional computer vision - a 

geometric viewpoint" 1993, MIT PRESS; 

US 2002/167512 A1 (BRODSKY TOMAS ET AL) 
November 14, 2002. 

The present invention suggests an alternate method 

30 which does not resort to manual initialisation of the resetting 
algorithm for each video sequence processed. It is moreover 
robust to the problem associated with contour detection, 
which is not the case of the methods described previously. 
Within the frameworl^ of the invention, the terms ground and 

35 scene are considered as equivalent. 
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Thus, the invention concerns, an automatic resetting 
method using electronic means intended for a geometric 
model of a scene over a picture of the scene, the model and 
the picture of the scene being stored in the memory of an 

5 electronic device in the form of pixel matrices, the scene 
Including fixed references with respect to the remainder of the 
scene, whereas the references may be specifically detected 
within the matrices, the picture being taken by a camera 
arranged in a given zone with respect to the ground in a 

10 location of the zone and according to a shot angle determined 
relative to the scene, the electronic means comparing the 
picture with the model having been adjusted In perspective by 
homography for superlmposltlon of the references. 
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The rough homography Hg is finally obtained by least 
square regression calculation carried out over all the pairs 
judged as non- aberrant. It should be noted that the 
calculation of Hg may be flne-tuned further by iterating the 
process described previously, new matching pairs of points 
being obtained by applying the homography Hg to the model- 
One may explain the calculation of the rough homography in 
the form of an algorithm with: 

• p1 : corresponding in the picture to a point p of the 
theoretic ground resetted by Hm (average 
homography) 

• p2 : contour point closest to pi 

1. For each point p belonging to the theoretic contour: 

a. p1=Hm(p) 

b. If p is a point belonging to a vertical line 
p2=p1+Dv(p1) 

c. If not p2= p1+Dh(p1) 

2. Robust calculation of the homography on the basis of 
the collection of the couples (p1,p2) found 

a. Perform n random draws of 4 couples of points 

b. For each draw: 

i. Calculate linearly the homography on the 
basis of the 4 couples 

ii. Calculate the medial error 

a. For the homography having given the minimal 

medial error 

i. Keep the non-aberrant couples (those 
whereof the absolute value of the residue is 
smaller than K times a). 

ii. Recalculate the rough homography Hg on 
the basis of all these couples 

c) the fine resetting 

The previous phase has therefore enabled to generate a 
matrix of rough homography Hg which is close to the final 
solution. The present step consists in fine-tuning the 
parameters of this homography in order to produce a fine 
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I.An automatic resetting method using electronic 
means Intended for a geometric model of a scene over a 
picture of the scene, the model and the picture of the scene 
being stored in the memory of an electronic device in the form 
of pixel matrices, the scene Including fixed references with 
respect to the remainder of the scene, whereas the references 
may be specifically detected within the matrices, the picture 
being tai<en by a camera arranged in a given zone with 
respect to the ground In a location of the zone and according 
to a shot angle determined relative to the scene, the 
electronic means comparing the picture with the model having 
been adjusted in perspective by homography for 
superlmposition of the references, 

characterised in that the electronic device calculates a fine 
homography function Hf for resetting Into three main phases: 
- - a first preliminary phase of determination of an average 
resetting homography consisting in determining an average 
homography function Hm applicable to the model with average 
adjustment over a sample of pictures of the scene taken 
previously. 

a second, rough resetting phase consisting after 
application of the average homography function to the 

model in determining a rough homography function Hg, said 
second rough resetting phase having following steps: 

- in a first step, an extraction process is applied to the 
picture enabling, according to detection criteria, to 
detect in the picture matrix of the pixels which may 
represent references of the scene and to form a first 
picture reference binary matrix Mrh Including horizontal 
contour points and a second picture reference binary 
matrix Mrv including vertical contour points, 

- in a second step, for each horizontal reference binary 
matrix Mrh. respectively vertical reference binary matrix 
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Mrv. a horizontal reference distance matrix, Mdh. 
respectively a vertical reference distance matrix Mdv 
including for each element of the matrix the distance 
value with respect to the closest reference according to 
the vertical line, respectively the horizontal line is 
calculated, 

- in a third step, all the reference lines of the model are 
applied the average homographic function Hm in order to 
produce a binary average adjusted matrix Mam which is 
compared with the vertical Mdv. respectively horizontal 
Mdh reference distance matrices, for pixel matching 
purposes, a homography function Hopt Is then calculated 
by regression with minimisation of the medial of the 
square of the distance between pairs of matched pixels. 

- in a fourth step, the pairs of pixels corresponding to 
non-aberrant matches are identified. 

- in a fifth step. Hopt Is adjusted by least square 
regression calculation over all the non-aberrant pixel 
pairs in order to produce the rough homography Hg, 

- - a third, fine resetting phase consisting after application of 
the rough homography function Hg to the model in determining 
a fine homography function Hf. 

2. A method according to claim 1, characterised in that 
in the preliminary step of determination of an average 
resetting homography, at least one sample picture is selected 
among a collection of pictures taken of the given location, the 
references on the sample plcture(s) are detected and an 
average homographic function Hm is calculated enabling 
superlmpositlon between the model subjected to the average 
homographic function and the sample plcture(s), 
superimposltion being reached for least error square 
minimization of the distance between reference points of 
sample picture(s) and the model subjected to the average 
homographic function. 
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3. A method according to claim 1 or 2. characterised In 
that in the second, rough resetting phase for the second step: 
in the horizontal reference distance matrix Mdh each element 
of said matrix specify the distance d in number of pixels 

5 relative to the reference line along a vertical axis, the 
distance values on the reference line and those of a column 
without any reference line pixel being nil, the distance values 
along the vertical line increasing in absolute value as the 
element moves away relative to the reference line, the 

10 distance values of the elements being of opposite signs on 
both sides of the reference line, 

in the vertical reference distance matrix Mdv each element of 
said matrix specifying the distance d in number of pixels 
relative to the reference line along a horizontal axis, the 

IS distance values on the reference line and those of a line 
without reference line pixel being nil, the distance values 
along the horizontal line increasing in absolute value as the 
element moves away relative to the reference line, of the 
elements being of opposite signs on both sides of the 

20 reference line, 

and that for pixels matching: 

with, for each pixel p(i,j) of the average adjusted matrix 
derived from a resetted pixel of the model belonging to a 
vertical reference line and positioned at the line i and at the 
25 column j of the average adjusted matrix Mam. the allocation of 
a corresponding pixel obtained by adding the value d in 1 and j 
of the vertical reference matrix Mdv to the value j, and 
matching the pixels ((i.j). (i,j+d)), 

with, for each pixel p(l,j) of the average adjusted matrix 
30 derived from a resetted pixel of the model belonging to a 
horizontal reference line and positioned at the line i and at 
the column j of the average adjusted matrix Mam. the 
allocation of a corresponding pixel obtained by adding the 
value d in i and j of the horizontal reference matrix Mdv to the 
35 value j, and matching the pixels ((i,j), (i+d,j)), 
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and that thee homography function Hopt calculated by 
regression with minimisation of the medial of the square of 
the distance between pairs of matched pixels, is carried out 
over n collections of four pairs of matched pixels. 

5 4. A method according to the claim 1. 2 or 3, 

characterised in that, in the fourth step of the second, rough 
resetting step, a pair of pixels corresponds to a non-aberrant 
match, if, for the pixel of the average adjusted matrix Mam of 
the match in question, the distance between the pixel 

10 matched by using the reference matrices Mdh, Mdv, and that 
obtained by the homography Hopt is smaller than or equal to a 
preset threshold. 

5. A method according to claim 3 or 4, characterised in 
that the reference detection criteria are chosen individually or 

IS in combination among: 

- a specific colour of the reference with respect to the 
remainder of the scene, 

- a specific tone of the reference with respect to the 
remainder of the scene, 

20 - a specific grey level of the reference with respect to the 
remainder of the scene, 

- a specific shape of the reference, notably a line, an angle 
between two lines crossing each other, a parallelism between 
two lines, 

25 - a specific orientation of the reference, 

- a line closest and parallel to an edge of the picture matrix. 

6. A method according to any of the the claims previous 
characterised in that the extraction process comprises a 
preliminary Cany-Deriche filtering step of the picture in order 

30 to obtain a gradient picture and that the process continues 
with the gradient picture. 

7. A method according to any of the previous claims, 
characterised In that in the third, fine resetting phase, the 
rough homography Hg is applied to the model and the result is 

35 compared to both horizontal and vertical distance matrices 
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with adjustment of the homography by a so-called Powei 
alternate single-dimension iterative minimisation method. 

8. A method according to any of the previous claims, 
characterised in that the pictures evolve with time according 

5 to sequences corresponding to different shot locations and/or 
angles and in that the electronic device comprises means 
enabling moreover to determine during the first, average 
resetting preliminary phase, as many average homography 
functions Hm as there are different shot locations and angles. 

10 9. A method according to any of the previous claims, 

characterised in that the phases and steps are implemented In 
the electronic means which are programmable logic units with 
a programme and that the programmable logic comprises a 
microprocessor or a digital signal processor (DSP) and. 

IS preferably, of the general-purpose or dedicated 
microcomputer type. 

10. A method according to any of the previous claims, 
characterised in that the scene is a sports ground including 
references in the form of delineating lines, notably a 

20 European or American "football" pitch or a tennis ground. 

11. Automatic resetting device using electronic means 
intended for a geometric model of a scene over a picture of 
the scene, the model and the picture of the scene being 
stored in the memory of an electronic device In the form of 

25 pixel matrices, the scene Including fixed references with 
respect to the remainder of the scene, whereas the references 
may be specifically detected within the matrices, the picture 
being talcen by a camera arranged In a given zone with 
respect to the ground in a location of the zone and according 

30 to a shot angle determined relative to the scene, the 
electronic means comparing the picture with the model having 
been adjusted in perspective by homography for 
superimpositlon of the references, 
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characterised in that it comprises means enabling to calculate 
a fine homography function Hf for resetting into three main 
phases: 

- - a first preliminary phase of determination of an average 
resetting homography consisting in determining an average 
homography function Hm applicable to the model with average 
adjustment over a sample of pictures of the scene taken 
previously, 

- - a second, rough resetting phase consisting after 
application of the average homography function to the 
model in determining a rough homography function Hg, said 
second rough resetting phase having following steps: 

• in a first step, an extraction process is applied to the 
picture enabling, according to detection criteria, to 
detect in the picture matrix of the pixels which may 
represent references of the scene and to form a first 
picture reference binary matrix Mrh including horizontal 
contour points and a second picture reference binary 
matrix Mrv including vertical contour points, 

- in a second step, for each horizontal reference binary 
matrix Mrh, respectively vertical reference binary matrix 
Mrv a horizontal reference distance matrix, Mdhi 
respectively a vertical reference distance matrix Mdv 
including for each element of the matrix the distance 
value with respect to the closest reference according to 
the vertical line, respectively the horizontal line, is 
calculated, 

- in a third step, all the reference lines of the model are 
applied the average homographic function Hm in order to 
produce a binary average adjusted matrix Mam which is 
compared with the vertical Mdv» respectively horizontal 
Mdh reference distance matrices, for pixel matching 
purposes, a homography function Hopt is then calculated 
by regression with minimisation of the medial of the 
square of the distance between pairs of matched pixels, 
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- in a fourth step, the pairs of pixels corresponding to 
non-aberrant matches are identified. 

- in a fifth step. IHopt is adjusted by ieast square 
regression calcuiation over all the non-aberrant pixel 

5 pairs in order to produce the rough homography Hg, 

- - a third, fine resetting phase consisting after application of 
the rough homography function to the model in determining 
a fine homography function Hf. 

12. A device according to claim 11, characterised in that 
10 the electronic means are of the general-purpose or dedicated 

microcomputer type. 

13. An information storage medium including a 
programme intended for operating the device of claim 1 1 . 

14. An information storage medium including a 
IS programme intended for operating the device of claim 1 1 and 

at least according to the method of claim 1 among the 
method-related claims 1 to 10. 
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